Apparatus and method for controlling motion-based user interface

ABSTRACT

A method and apparatus for controlling a motion-based user interface are provided. The apparatus includes a touch screen for displaying an image and for receiving input of a user touch on at least one spot of the touch screen, a sensor unit for sensing a motion of the apparatus, a motion calculator for calculating a degree of the motion of the apparatus, when sensing the motion of the apparatus at the sensor unit, and a controller for, when the at least one spot is touched on the touch screen, determining the number of touched spots, for receiving information about the degree of the motion of the apparatus from the motion calculator, and for determining whether to change the size of the image or the position of the image according to the number of touched spots.

PRIORITY

This application is a continuation application of prior application Ser.No. 13/879,203, filed on Apr. 12, 2013, which will issue as U.S. Pat.No. 9,588,613 on Mar. 7, 2017, and which is a National Stage applicationunder 35 U.S.C. §371 of an International application filed on Oct. 13,2011 and assigned application No. PCT/KR2011/007628, and claimed thebenefit under 35 U.S.C § 119(a) of a Korean patent application filed onOct. 14, 2010 in the Korean Intellectual Property Office and assignedSerial No. 10-2010-0100435, the entire disclosure of which is herebyincorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a user interface. More particularly,the present invention relates to an apparatus and method for controllingan interface according to a motion that a mobile terminal has made.

2. Description of the Related Art

Currently, electronic devices such as TVs, MP3 players, PortableMultimedia Players (PMPs), smart phones, and the like are equipped witha variety of input/output devices in order to enable a user toconveniently control the electronic device.

Among these input/output devices, the utilization of smart phones hasrecently increased exponentially. The proportion of devices using touchpanels is rapidly growing in the market of mobile terminals, including aportable phone, a smart phone, and a laptop. As touch screen panels areexpected to gain more popularity, the market for touch screens formobile terminals will be rapidly boosted. Touch screen panels are alsowidely used in electronic appliances such as TVs or refrigerators. Themarket of electronic appliances will likely rank second in adoptingtouch screen panels, following the mobile terminal market.

Recently, extensive research has been made on recognition of a user'sintention and action based on visual information, for naturalinteraction between a user and a touch screen. Typically, auser-friendly user interface is configured to recognize a gesture inputby a finger or a touch pen. The trend of user interfaces is now shiftingfrom an interface that operates according to a single finger-basedsingle touch input on a touch screen to an interface that operatesaccording to a multi-finger-based multi-touch input on a touch screen.

A touch screen is includes a plane (e.g., substrate) for sensing aninput and a plane serving as a display. Therefore, a user's intentioncan be analyzed and perceived from a multi-touch input on the touchscreen and the analysis and perception result may be output on the touchscreen. A multi-touch-based user interface is designed in such a mannerthat the number of finger touches/pen touches on the touch screen and anassociated operation are recognized and an associated command isexecuted. The interior structure of a mobile terminal supportingmulti-touch input is described below.

FIG. 1 is a block diagram of a mobile terminal supporting multi-touchinput according to the related art.

Referring to FIG. 1, the mobile terminal includes an output unit 101, acontroller 103, a memory 105, and an input unit 107.

The output unit 101 outputs an image (such as a drawing or a Web page)on a touch screen. The image such as a drawing or a Web page is storedin the memory 105. When the image is output on the touch screen, a usermay enlarge or shrink the image using the input unit 107 according touser selection. An intended part of the image displayed on the touchscreen may be enlarged or contracted by touching the intended part withtwo fingers simultaneously. A touch pen may be used instead of fingers.Upon input of multiple touches through the input unit 107, thecontroller 103 controls the output unit 101 to display the multi-touchedarea enlarged or shrunk on the touch screen.

An exemplary operation performed upon input of multiple touches in themobile terminal having the configuration illustrated in FIG. 1 isdescribed below.

FIGS. 2A and 2B illustrate an exemplary operation performed upon inputof multiple touches according to the related art. The operation isspecifically for enlarging a specific area by multiple touches on atouch screen.

Referring to FIG. 2A, a user touches the touch screen 201 with twofingers 205 in a pinching motion to enlarge an image of a car 203displayed on a touch screen 201. The enlarged area may be differentdepending on a position at which the two fingers 205 touch. Referring toFIG. 2B, the car image 203 is enlarged by spreading out the two fingers205 on the touch screen 201. The degree to which the car image 203 isenlarged may depend on the distance between the two fingers 205 spreadapart from each other.

While only the operation for enlarging an image is illustrated in FIGS.2A and 2B, the enlarged image may also be shrunk using the two fingers205.

According to the related art, the zoom-in or zoom-out ratio of apredetermined part of the touch screen may be adjusted only by pressinga predefined zoom-in or zoom-out key or making multiple touches. When auser wants to move from one page to another page or from one part toanother part on the same page, the user is supposed to press apredefined move key or touch and drag an area of the touch screen. Thismeans that for continuous zoom-in and zoom-out, the user should makeconsecutive key inputs or continuous multiple touches. In addition, tomove from a specific part of a page to another part of the same page,the user should input keys successively or make continuoustouches-and-drags, which the user may find inconvenient. Therefore, itis difficult to quickly and accurately perform a user-intended operationsimply with an input on the touch screen or a gesture drawn on the touchscreen.

SUMMARY OF THE INVENTION

Aspects of the present invention are to address at least theabove-mentioned problems and/or disadvantages and to provide at leastthe advantages described below. Accordingly, an aspect of the presentinvention is to provide an apparatus and method for controlling a userinterface so that an image may be enlarged, shrunk, and moved based on amotion.

In accordance with an aspect of the present invention, an apparatus forcontrolling a motion-based user interface is provided. The apparatusincludes a touch screen for displaying an image and for receiving inputof a user touch on at least one spot of the touch screen, a sensor unitfor sensing a motion of the apparatus, a motion calculator for, whensensing the motion of the apparatus at the sensor unit, calculating adegree of the motion of the apparatus, and a controller for, when the atleast one spot is touched on the touch screen, determining the number oftouched spots, for receiving information about the degree of the motionof the apparatus from the motion calculator, and for determining whetherto change the size of the image or the position of the image accordingto the number of touched spots.

In accordance with another aspect of the present invention, a method forcontrolling a motion-based user interface is provided. The methodincludes displaying an image on a touch screen, when at least one spotis touched on the touch screen, determining a number of the at least onetouched spot, determining whether to change the size of the image or theposition of the image according to the number of touched spots, andchanging the size of the image or the position of the image according toa motion of an apparatus including the touch screen.

According to the apparatus and method of the present invention, the usermay easily enlarge, shrink, and move an image using a motion.

BRIEF DESCRIPTION OF DRAWINGS

The above and other objects, features and advantages of certainexemplary embodiments of the present invention will be more apparentfrom the following description taken in conjunction with theaccompanying drawings, in which:

FIG. 1 is a block diagram of a mobile terminal supporting multi-touchinput according to the related art;

FIGS. 2A and 2B illustrate an exemplary operation that is performedbased on a multi-touch input according to the related art;

FIG. 3 is a block diagram of a mobile terminal for controlling amotion-based user interface according to an exemplary embodiment of thepresent invention;

FIGS. 4A-4C are exemplary views illustrating motions of a mobileterminal according to an exemplary embodiment of the present invention;

FIGS. 5A-5C are exemplary views illustrating motions of a mobileterminal according to an exemplary embodiment of the present invention;

FIGS. 6A and 6B are exemplary views illustrating an operation forenlarging or shrinking an image according to an exemplary embodiment ofthe present invention;

FIGS. 7A and 7B are exemplary views illustrating an operation for movingan image according to an exemplary embodiment of the present invention;

FIGS. 8A and 8B are exemplary views illustrating operations forcontrolling a motion-based user interface using predetermined keysaccording to an exemplary embodiment of the present invention; and

FIG. 9 is a flowchart illustrating an operation for controlling amotion-based user interface according to an exemplary embodiment of thepresent invention.

Throughout the drawings, should be noted that like reference numbers areused to depict the same or similar elements, features and structures.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of exemplaryembodiments of the invention as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding, but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill in the art will recognize thatvarious changes and modifications of the embodiments described hereincan be made without departing from the scope and spirit of theinvention. In addition, descriptions of well-known functions andconstructions may be omitted for clarity and conciseness.

FIG. 3 is a block diagram of a mobile terminal for controlling amotion-based user interface according to an exemplary embodiment of thepresent invention.

Referring to FIG. 3, the mobile terminal includes an output unit 301, acontroller 303, a memory 305, an input unit 307, a motion calculator309, and a sensor unit 311.

The output unit 301 outputs an image such as a photo or a Web page on atouch screen. The image is stored in the memory 305. When needed,another image stored in the memory 305 may be used or an intended photomay be captured using a camera module equipped in the mobile terminal.However, it is to be understood that an image may refer to any object(or portion thereof) displayed on the touch screen.

The input unit 307 receives information about the number of spotstouched on the touch screen, a drag made on the touch screen, and thelike. The motion calculator 309 calculates the movement direction orangle of the mobile terminal and transmits information about thecalculated movement direction or angle to the controller 303. The mobileterminal may move up, down, to the left, and to the right. In addition,the mobile terminal may move in a horizontal direction toward the useror in a horizontal direction away from the user. The angle at which themobile terminal has moved is calculated to be an angle at which themobile terminal is inclined with respect to a current position set as 0degree.

The motion calculator 309 uses the sensor unit 311 to calculate themovement direction or angle of the mobile terminal. The motioncalculator 309 calculates a direction or angle in which the mobileterminal has moved using a gyroscope sensor from among one or moresensors of the sensor unit 311. The gyroscope sensor is a sensor adaptedto sense rotations in addition to the functions of an accelerationsensor (i.e., accelerometer) and thus to track motions on 6 axes.Accordingly, compared to the acceleration sensor, the gyroscope sensorcan sense more precise motions. The gyroscope sensor senses height androtational inclination as well as acceleration and deceleration.

The controller 303 receives information about the number of touchedspots and information about a touch and drag on the touch screen fromthe input unit 307, receives information about a calculated motion ofthe mobile terminal from the motion calculator 309, and controls animage displayed on the touch screen in a different manner according tothe number of touched spots.

If two or more spots are touched in an area of the touch screen, thecontroller 303 determines the multi-touch input to be a command forenlarging or shrinking an image displayed on the touch screen andenlarges or shrinks the image according to information about a change inthe motion of the mobile terminal received from the motion calculator309. For example, when two spots are touched in an area of the touchscreen and the mobile terminal is brought closer to the user at anunchanged angle or the touch screen of the mobile terminal is inclinedforward with respect to the user, the image is enlarged. On the otherhand, when the mobile terminal recedes from the user at an unchangedangle or the touch screen of the mobile terminal is reclined backwardfrom the user, the image is shrunk. This method has the same effects asachieved in the multi-touch scheme according to the related art in whichan image is enlarged or shrunk using two fingers, which will be apparentfrom a description of FIGS. 4A, 4B and 4C.

FIGS. 4A, 4B and 4C are exemplary views illustrating motions of themobile terminal according to an exemplary embodiment of the presentinvention. In FIGS. 4A, 4B and 4C, it is assumed that a user touches twospots on the touch screen.

Referring to FIG. 4A, when the user pulls the mobile terminal in the−Z-axis direction with the current angle of the mobile terminalunchanged, facing the touch screen of the mobile terminal, whiletouching any two spots on the touch screen of the mobile terminal, adisplayed image is enlarged. When the user pushes the mobile terminal inthe Z-axis direction, facing the touch screen of the mobile terminal,the displayed image is shrunk. FIG. 4B is a side view of the mobileterminal, illustrating a path in which the mobile terminal is moved. InFIG. 4B, the mobile terminal is moved only in the Z-axis or −Z-axisdirection without a change in its angle. FIG. 4C illustrates a variationin the inclination of the mobile terminal, with the Y and −Y axes set asa rotational axis. An image displayed on the touch screen may beenlarged or shrunk by inclining the mobile terminal forward or backwardwith respect to the user.

If the user touches one spot in an area of the touch screen, thecontroller 303 determines the single-touch input to be a command formoving an image displayed on the touch screen and moves the image up,down, to the left, or to the right according to information about achange in the motion of the mobile terminal received from the motioncalculator 309. For example, if the user moves the mobile terminal tothe left or rotates the mobile terminal to the left, while touching onespot in an area of the touch screen, a displayed image is moved to theleft. The same effect as with a scheme for moving an image in anintended direction by touching and dragging an area of the touch screenis achieved, which will be apparent from FIG. 5.

FIGS. 5A, 5B and 5C illustrate motions of a mobile terminal according toan exemplary embodiment of the present invention. In FIGS. 5A, 5B and5C, it is assumed that the user touches one spot on the touch screen.

Referring to FIG. 5A, when the user moves the mobile terminal in theX-axis, −X-axis, Y-axis or −Y-axis direction, while touching one spot onthe touch screen of the mobile terminal with the current angle of themobile terminal unchanged, an image displayed on the touch screen movesin the direction in which the mobile terminal is moved. FIG. 5Billustrates changes in the rotation of the mobile terminal, with the Zand −Z axes set as a rotational axis and FIG. 5C illustrates changes inthe rotation of the mobile terminal, with the Y and −Y axes set as arotational axis. If the Z and −Z axes are set as a rotational axis andthe mobile terminal is rotated to the right or to the left on the Y and−Y axes, the displayed image may move sideways. On the other hand, ifthe Y and −Y axes are set as a rotational axis and the touch screen ofthe mobile terminal is inclined or reclined on the Y and −Y axes, thedisplayed image may move up or down.

If the Y and −Y axes are set as a rotational axis, the displayed imagemay be transformed differently according to the number of touched spotson the touch screen. In FIG. 4C, the image is enlarged or shrunk,whereas in FIG. 5C, the image is moved up or down.

As is done according to the related art, the controller 303 may alsocontrol an image displayed on the touch screen in correspondence with agesture such as a drag that may follow a touch of at least one spot inan area of the touch screen. For example, if one spot is touched in anarea of the touch screen and the touched spot is dragged to the left,the display image may be moved to the left or the next image may bedisplayed. If two spots are touched on the touch screen, the image maybe enlarged or shrunk according to the multi-touch scheme of the relatedart.

A description of an operation for controlling a displayed image isdescribed below, with reference to the mobile terminal of FIG. 3.

FIGS. 6A and 6B are exemplary views illustrating an operation forenlarging or shrinking an image according to an exemplary embodiment ofthe present invention.

Referring to FIG. 6A, the user pulls the mobile terminal, grabbing thetouch screen so that two spots in an area of the touch screen aretouched, while viewing the touch screen. A displayed image is enlargedin correspondence with the pulling motion. FIG. 6B is a side viewillustrating the user pulling the mobile terminal. It is noted that theimage is enlarged simply by bringing the mobile terminal closer to theuser as illustrated in FIGS. 4A and 4B without inclining the mobileterminal forward with respect to the user.

While not shown in FIGS. 6A and 6B, when the user pushes the mobileterminal backward, grabbing the touch screen so that two spots in anarea of the touch screen are touched, while viewing the touch screen,the displayed image may be shrunk. The function of enlarging an image bypulling the mobile terminal and shrinking an image by pushing the mobileterminal may be changed by a user setting. For instance, the functionmay be changed in such a manner that an image is shrunk by pulling themobile terminal and enlarged by pushing the mobile terminal.

The zoom-in or zoom-out ratio of an image may be set to be proportionalto a rotational angle measurement in each axis direction illustrated inFIGS. 4A to 5C, using the gyroscope sensor included in the sensor unit311 of the mobile terminal displaced according to a user motion. In thesituation where this image zoom-in or zoom-out function is available,the user may enlarge or shrink an image according to the conventionalmulti-touch scheme. While the multi-touch-based zoom-in or zoom-outfunction according to the related art is in progress, the motion-basedimage zoom-in or zoom-out function may be deactivated to avoid mutualinterference between the functions according to the related art and theproposed zoom-in or zoom-out functions according to exemplaryembodiments of the present invention.

If only one spot is touched or no touch is made on the touch screen, theimage zoom-in and zoom-out operation is deactivated in the presentinvention.

FIGS. 7A and 7B are exemplary views illustrating an operation for movingan image according to an exemplary embodiment of the present invention.

Referring to FIG. 7A, the user moves the mobile terminal from the leftto the right, viewing the touch screen, while grabbing the mobileterminal so that one spot is touched in an area of the touch screen. Animage displayed on the touch screen moves from the right to the left incorrespondence with the motion of the mobile terminal.

Referring to FIG. 7B, focusing on the touch screen, the left touchscreen displays an image prior to the movement, and the right touchscreen displays an image output when the mobile terminal moves from theleft to the right. As the image moves, an image hidden at the right sideoutside the screen appears on the screen. In the case where a pluralityof images are arranged, the images may sequentially show up on thescreen by moving the mobile terminal.

The distance for which an image moves may be set to be proportional to arotational angle measurement in each axis direction illustrated in FIGS.4A to 5C, using the gyroscope sensor included in the sensor unit 311 ofthe mobile terminal displaced according to a user motion. In thesituation where this image shift function is available, the user maymove an image by touching one spot and dragging the touch on the touchscreen. During the dragging function, the motion-based image shiftfunction may be deactivated to avoid mutual interference between thedragging function and the image shift function of the present invention.

According to exemplary embodiments of the present invention, tf only onespot is touched or no touch is made on the touch screen, the image shiftoperation is deactivated.

While it is described according to according to exemplary embodiments ofthe present invention that a displayed image can be controlled in adifferent manner according to the number of touched spots in an area ofthe touch screen, the same object may be achieved by assigning specifickeys and controlling the displayed image according to the number ofpressed keys, which is described below.

FIGS. 8A and 8B are exemplary views illustrating an operation forcontrolling a motion-based user interface using assigned specific keysaccording to an exemplary embodiment of the present invention.

Referring to FIG. 8A, when the user presses buttons of digits 1 and 3 atthe same time or at a predetermined interval on a keypad while executingan image viewer (or other application), the mobile terminal may be setto recognize the motion as a command for enlarging or shrinking adisplayed image. According to a subsequent motion of the mobileterminal, the displayed image may be enlarged or shrunk.

Referring to FIG. 8B, when the user presses either of the buttons ofdigits 1 and 3 on the keypad while executing the image viewer, themobile terminal may be set to recognize the motion as a command forshifting a displayed image. The keypad may be replaced with buttons thatoperate based on user touches (e.g., a key displayed on the touchscreen).

FIG. 9 is a flowchart illustrating an operation for controlling amotion-based user interface according to an exemplary embodiment of thepresent invention.

Referring to FIG. 9, upon receipt of a user input, the mobile terminalexecutes an image viewer to display an image such as a photo or Web pagein step 901. The image is stored in the memory 305. If the mobileterminal is wireless Internet-enabled, the mobile terminal may display aWeb page without executing the image viewer. In step 903, the mobileterminal determines whether a touch input has been sensed through theinput unit 307. If a touch input has not been sensed, the mobileterminal continues executing the image viewer in step 901.

Upon sensing a touch input, the mobile terminal counts the number ofspots touched on the touch screen and controls a displayed image in adifferent manner according to the number of touched spots in step 905.If two or more spots have been touched, the mobile terminal determinesto enlarge or shrink the image and starts to sense a motion of themobile terminal in step 907. In step 907, if the mobile terminal movesforward or backward or is inclined forward with respect to the user orreclined backward as illustrated in FIGS. 4A, 4B and 4C, the mobileterminal calculates a degree to which the mobile terminal moves up,down, to the left, to the right, back, or forth or a degree to which theapparatus is inclined forward, reclined backward, rotated to the left,or rotated to the right, using a gyroscope sensor included in the sensorunit and enlarges or shrinks the image in correspondence with thecalculated degree. The motion of the mobile terminal is sensed by thegyroscope sensor of the sensor unit 311 and the distance for which themobile terminal has moved or the degree to which the mobile terminal hasbeen inclined is calculated by the motion calculator 309. In addition,two or more buttons may be assigned and pressed instead of two spots soas to enlarge or shrink the image.

If a single spot has been touched, the mobile terminal determines toshift the image in step 909 and starts to sense a motion of the mobileterminal. When the mobile terminal moves up, down, to the left, or tothe right, or is inclined forward, reclined backward, rotated to theleft, or rotated to the right as illustrated in FIGS. 5A, 5B and 5C, themoved distance or inclination or rotation degree of the mobile terminalis calculated and the image is shifted in a direction corresponding tothe calculated degree. As described above, the motion of the mobileterminal is sensed by the gyroscope sensor of the sensor unit 311 andthe distance for which the mobile terminal has moved or the degree towhich the mobile terminal has been inclined is calculated by the motioncalculator 309. In addition, a specific button may be assigned andpressed instead of one spot so as to shift the image.

While the invention has been shown and described with reference tocertain exemplary embodiments thereof, it will be understood by thoseskilled in the art that various changes in form and details may be madetherein without departing from the spirit and scope of the invention asdefined by the appended claims and their equivalents.

What is claimed is:
 1. An electronic device for displaying an image,comprising: a display configured to display an image; and a processorconfigured to: if a number of touched spots on the display is a firstvalue, determine a movement of the electronic device as a command formoving the image, determine a shift degree of the image corresponding tothe movement of the electronic device, and shift the image based on theshift degree and control the display to display the shifted image. 2.The electronic device of claim 1, further comprising: one or moresensors configured to generate one or more sensing values related to themovement of the electronic device; wherein the processor is furtherconfigured to determine a degree of the movement based on the one ormore sensing values.
 3. The electronic device of claim 1, wherein theprocessor is further configured to: if the electronic device moves fromthe left to the right, shift the image from the right to the left. 4.The electronic device of claim 1, wherein the processor is furtherconfigured to: if the electronic device moves from the right to theleft, shift the image from the left to the right.
 5. The electronicdevice of claim 1, wherein the processor is further configured to: ifthe electronic device moves up, shift the image down.
 6. The electronicdevice of claim 1, wherein the processor is further configured to: ifthe electronic device moves down, shift the image up.
 7. The electronicdevice of claim 1, wherein the processor is further configured to: ifthe electronic device moves from the left to the right, control thedisplay to display a next image from among a plurality of images in anarrangement order of the plurality of images.
 8. The electronic deviceof claim 1, wherein the processor is further configured to: if theelectronic device moves from the right to the left, control the displayto display a previous image from among a plurality of images in anarrangement order of the plurality of images.
 9. The electronic deviceof claim 1, wherein the processor is further configured to: if thenumber of the touched spots on the display is a second value, determinethe movement of the electronic device as a command for zooming-in orzooming-out the image, and zoom-in or zoom-out the image based on amovement degree of the electronic device.
 10. A method for displaying animage by an electronic device, comprising: displaying an image; if anumber of touched spots on a display of an electronic device is a firstvalue, determining a movement of the electronic device as a command formoving the image; determining a shift degree of the image correspondingto the movement of the electronic device; and shifting the image basedon the shift degree and displaying the shifted image.
 11. The method ofclaim 10, further comprising: generating one or more sensing valuesrelated to the movement of the electronic device; and determining adegree of the movement based on the one or more sensing values.
 12. Themethod of claim 10, wherein, if the electronic device moves from theleft to the right, the image is shifted from the right to the left. 13.The method of claim 10, wherein, if the electronic device moves from theright to the left, the image is shifted from the left to the right. 14.The method of claim 10, wherein, if the electronic device moves up, theimage is shifted down.
 15. The method of claim 10, wherein if theelectronic device moves down, the image is shifted up.
 16. The method ofclaim 10, wherein, if the electronic device moves from the left to theright, a next image is displayed from among a plurality of images in anarrangement order of the plurality of images.
 17. The method of claim10, wherein, if the electronic device moves from the right to the left,a previous image is displayed from among a plurality of images in anarrangement order of the plurality of images.
 18. The method of claim10, further comprising: if the number of the touched spots on thedisplay is a second value, determining the movement of the electronicdevice as a command for zooming-in or zooming-out the image; andzooming-in or zooming-out the image based on a movement degree of theelectronic device.